Preserving engagement state based on contextual signals

ABSTRACT

A method for dynamically changing a graphical user interface element occurs in response to detecting that a temporal user interface element displayed on a user interface of user device. The method includes receiving, at the user device, a contextual signal characterizing a state of a user. The method further includes determining, by the user device, that the contextual signal characterizing the state of the user is indicative of the user intending to interact with a temporal user interface element. The methods also include, in response to determining that the contextual signal characterizing the state of the user is indicative of the user intending to interact with a temporal user interface element, modifying a respective state of the temporal user interface element displayed on the user interface of the user device.

CROSS REFERENCE TO RELATED APPLICATIONS

This U.S. patent application is a continuation of, and claims priorityunder 35 U.S.C. § 120 from, U.S. patent application Ser. No. 17/444,656,filed on Aug. 6, 2021. The disclosure of this prior application isconsidered part of the disclosure of this application and is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

This disclosure relates to preserving an engagement state based oncontextual signals.

BACKGROUND

A speech-enabled environment (e.g., home, workplace, school, automobile,etc.) allows a user to speak a query or a command out loud to acomputer-based system that fields and answers the query and/or performsa function based on the command. The speech-enabled environment can beimplemented using a network of connected microphone devices distributedthrough various rooms or areas of the environment. These devices may usehotwords to help discern when a given utterance is directed at thesystem, as opposed to an utterance that is directed to anotherindividual present in the environment. Accordingly, the devices mayoperate in a sleep state or a hibernation state and wake-up only when adetected utterance includes a hotword. Once awake, the devices canproceed to perform more expensive processing such as full on-deviceautomated speech recognition (ASR) or server-based ASR.

SUMMARY

One aspect of the disclosure provides a computer-implemented method fordynamically changing a graphical user interface element. Thecomputer-implemented method, when executed by data processing hardware,causes the data processing hardware to perform operations. Theoperations occur in response to detecting that a temporal user interfaceelement displayed on a user interface of user device. The operationsinclude receiving, at the user device, a contextual signalcharacterizing a state of a user. The operations further includedetermining, by the user device, that the contextual signalcharacterizing the state of the user is indicative of the user intendingto interact with a temporal user interface element. The operations alsoinclude, in response to determining that the contextual signalcharacterizing the state of the user is indicative of the user intendingto interact with a temporal user interface element, modifying arespective state of the temporal user interface element being displayedon the user interface of the user device.

Another aspect of the disclosure provides a system for dynamicallychanging a graphical user interface element. The system includes dataprocessing hardware and memory hardware in communication with the dataprocessing hardware. The memory hardware stores instructions that whenexecuted on data processing hardware cause the data processing hardwareto perform operations. The operations occur in response to detectingthat a temporal user interface element displayed on a user interface ofuser device. The operations include receiving, at the user device, acontextual signal characterizing a state of a user. The operationsfurther include determining, by the user device, that the contextualsignal characterizing the state of the user is indicative of the userintending to interact with a temporal user interface element. Theoperations also include, in response to determining that the contextualsignal characterizing the state of the user is indicative of the userintending to interact with a temporal user interface element, modifyinga respective state of the temporal user interface element displayed onthe user interface of the user device

Implementations of either aspect of the disclosure may include one ormore of the following optional features. In some implementations, thestate of the user includes an engagement state indicative of the userattempting to engage or engaging with the temporal user interfaceelement displayed on the user interface of the user device. In someexamples, modifying the respective state of the oral user interfaceelement includes increasing or suspending a timeout duration of thetemporal user interface element. In some configurations, the state ofthe user includes a disengagement state indicative of a user disengagingwith the temporal user interface element displayed on the user interfaceof the user device. In these configurations, in response to determiningthat the contextual signal characterizing the state of the user includesthe disengagement state, modifying the respective state of the temporaluser interface element may include removing the temporal user interfaceelement prior to the expiration of a timeout duration for the temporaluser interface element. In these configurations, in response todetermining that the contextual signal characterizing the state of theuser includes the disengagement state, modifying the respective state ofthe temporal user interface element may include decreasing a timeoutduration of the temporal user interface element. Some examples of thecontextual signal include a user proximity signal indicating a proximityto the user relative to the user device, a presence detection signalindicating a presence of the user within the field of view of a sensorassociated with the user device, and an attention detection signalindicating an attention of the user with respect to the user device. Thetemporal user-interface element may represent an action specified by aquery detected in streaming audio captured by the user device.Optionally, the operations further include, in response to determiningthat the contextual signal characterizing the state of the user isindicative of the user intending to interact with the temporaluser-interface element and prior to modifying the respective state ofthe temporal user-interface element being displayed on the userinterface of the user device, determining that the respective state ofthe temporal user-interface element has failed to have been previouslymodified a threshold number of times within a period of time.

In some examples of either aspect of the disclosure, the contextualsignal includes a presence detection signal indicating a presence of theuser within the field of view of a sensor associate with the userdevice. Here, determining that the contextual signal characterizing thestate of the user is indicative of the user intending to interact with atemporal user interface element includes determining that the presencedetection signal indicates that the presence of a user within the fieldof view of the sensor has changed from not present to present andmodifying the respective state of the temporal user interface elementbeing displayed on the user interface includes increasing a timeoutduration or suspending the timeout duration of the temporal userinterface element.

In some implementations of either aspect of the disclosure, thecontextual signal includes a user proximity signal indicating aproximity of the user relative to the user device. Here, determiningthat the contextual signal characterizing the state of the user isindicative of the user intending to interact with the temporaluser-interface element includes determining that the user-proximitysignal indicates that the proximity of the user relative to the userdevice has changed to be closer to the user device and modifying therespective state of the temporal user-interface element being displayedon the user interface includes increasing a timeout duration orsuspending the timeout duration of the temporal user-interface element.

In some configurations of either aspect of the disclosure, thecontextual signal includes an attention detection signal indicating anattention of the user with respect to the user device. Here, determiningthat the contextual signal characterizing the state of the user isindicative of the user intending to interact with the temporaluser-interface element includes determining that the attention detectionsignal indicates that the attention of the user has changed to focus onthe user device and modifying the respective state of the temporaluser-interface element being displayed on the user interface includesincreasing a timeout duration or suspending the timeout duration of thetemporal user-interface element.

The details of one or more implementations of the disclosure are setforth in the accompanying drawings and the description below. Otheraspects, features, and advantages will be apparent from the descriptionand drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic view of an example speech-enabled environment.

FIGS. 2A and 2B are schematic views of example preserver for thespeech-enabled environment of FIG. 1 .

FIG. 3 is a schematic view of example speech-enabled environments usingthe preserver.

FIG. 4 is a flow chart of an example arrangement of operations for amethod of changing a state of a graphical user-interface element.

FIG. 5 is a schematic view of an example computing device that may beused to implement the systems and methods described herein.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

A voice-enabled device (e.g., a user device executing a voice assistant)allows a user to speak a query or a command out loud and field an answerfor the query and/or perform a function based on the command. Often,when the voice-enabled device responds to the query, the voice-enableddevice generates a visual response that represents an action requestedby the query. For instance, a user of the voice-enabled device speaks anutterance requesting that the voice-enabled device plays a particularsong on a music streaming application of the voice-enabled device. Inresponse to this request by the user, the voice-enabled device displaysa graphical element on a display associated with the voice-enableddevice that indicates that the voice-enabled device is playing therequested song on the music streaming application associated with amusic streaming service.

In certain situations, the graphical element is temporary or temporal innature such that the graphical element is removed from the display ofthe voice-enabled device after a particular amount of time. That is, thegraphical element may have a set timeout value where the timeout valuerefers to a duration that the graphical element will visually existbefore being removed (i.e., timing-out). For example, the graphicalelement is a temporary notification that visually indicates the actionthat the voice-enabled device is performing or can perform in responseto the query by the user.

In some examples, the voice-enabled device does not perform the actionin response to the query until the user associated with thevoice-enabled device somehow confirms that the user wants the action tobe performed. Here, the confirmation may be a verbal confirmation (e.g.,“Yes”) such as a speech input, a motion confirmation such as a gesture,or tactile confirmation such as the user tapping the display of thevoice-enabled device to affirm performance of the action. In using aconfirmation approach, the confirmation approach may prevent avoice-enabled device from inadvertently performing an action that is notdesired by the user. Moreover, this approach allows the voice-enableddevice to suggest one or more actions that the voice-enabled deviceperceived that the user requested and for the user to select or indicateif any of the actions were actually requested by the user (i.e., anyactions that the user intended for the voice-enabled device to perform).

In some implementations, the voice-enabled device suggests an action inresponse to a query by the user rather than automatically performing theaction because the voice-enabled device lacks confidence that the useractually requested that the action be performed. For instance, one useris having a conversation with another user in proximity to thevoice-enabled device and the voice-enabled device perceives that someaspect of the conversation was a query for the voice-enabled device toperform an action. Yet because this is a side speech conversation thatis not directed to the voice-enabled device, the voice-enabled devicemay not determine the action with a particular level of confidence toautomatically perform the action. Here, due to a lower confidence thatdoes not satisfy automatic performance of the action, the voice-enableddevice instead displays a graphical element that inquires whether theuser wants to perform the detected action. For instance, thevoice-enabled device displays a prompt that states, “Do you want me toplay Thriller, by Michael Jackson?” This prompt therefore gives the userthe ability to confirm that is what the user intended (i.e., aselectable “yes” button in the prompt) or gives the user the ability toindicate that was not what the user intended (i.e., a selectable “no”button in the prompt). Here, the prompt may be setup as a temporalgraphical element which allows the user to ignore the prompt or fail tobe aware of the prompt and not cause the action suggested by the promptto be performed. In other words, if the voice-enabled device does notreceive any input from the user regarding the prompt, the prompt andtherefore the suggested action will timeout and not occur.

Unfortunately, situations occur where the user did request that aparticular action be performed by the voice-enabled device, but the useris unable to provide confirmation that the user wants the action to beperformed. To illustrate, a voice-enabled device may be located in akitchen of the user's home. When the user returns home from the grocerystore, the user then begins to carry groceries into the kitchen in aseries of trips from his or her car. While in the kitchen initially, theuser requests that the voice-enabled device play a local radio stationthat the user was listening to on the way home from the grocery store.In this scenario, although the user explicitly requested that thevoice-enabled device play the local radio station, the user may havebeen moving and turning such that the voice-enabled device did notconfidently detect the command. Due to this lack of confidence, thevoice-enabled device generates a prompt that inquires whether the userwants the voice-enabled device to play the local radio station. Eventhough this suggested action is indeed what the user requested, the usermay fail to see the prompt and become frustrated that the voicecapabilities of the voice-enabled device are underperforming.Alternatively, the user may return from a trip to the car with arms fullof groceries only to notice that the local radio station is not playingand a prompt is present as a temporal graphical element on the displayof the voice-enabled device. Yet before the user can set the groceriesdown or has a chance to verbally respond, the prompt may disappear(i.e., timeout). In this example, prior to the prompt/suggested actiontiming out, the user may have been moving toward voice-enabled device totap yes on the prompt.

For situations like the grocery example, a state of the temporalgraphical element may benefit from being aware of a state of the user.That is, in the grocery example, the user was moving toward thevoice-enabled device as the prompt disappeared (i.e., timed-out).Meaning that, if some system associated with voice-enabled devicerecognized the state of the user as attempting to interact with thevoice-enabled device, that system may inform the voice-enabled device(or some system thereof) to extend the timeout duration of the temporalgraphical element (e.g., the prompt asking whether the user wanted toplay the local radio station). Accordingly, if the timeout duration wasextended due to the state of the user, the user may have enough time toinput confirmation of the action at the voice-enabled device; resultingin the voice-enabled device successfully playing the local radio stationto the enjoyment of the user.

In contrast, if, in the grocery example, the user came into the kitchencarrying groceries and talking on his or her cell phone saying, “I justheard this great song playing on the local radio station,” the temporalgraphical element may also benefit from being aware of the state of theuser to accelerate its timeout or timeout immediately (i.e., change thestate of the temporal element). For instance, in this alternative to theexample, the user may stare at the voice-enabled device noticing thatthe voice-enabled device generated a prompt to play the local radiostation and then walk away to continue bringing in groceries. Here, thevisual recognition of the user (i.e., the stare) combined with thenwalking away may inform the voice-enabled device to timeout the promptimmediately due to the implicit lack of interest in the suggested actionby the user.

In order for the temporal element to be aware of the state of the user,one or more systems associated with the voice-enabled may be configuredto analyze contextual signals to determine whether one or more of thesesignals indicate that the state of the user should impact the state ofthe temporal element. In other words, if contextual signal(s)characterize the state of the user as attempting to positively interact(i.e., an engagement state) with the voice-enabled device (e.g.,attempting to engage), the state of the temporal element may be modifiedto accommodate for the positive interaction (e.g., the timeout isextended or suspended entirely). On the other hand, if contextualsignal(s) characterize the state of the user as attempting to negativelyinteract (i.e., a disengagement state) with the voice-enabled device(e.g., to not engage), the state of the temporal element may be modifiedto accommodate for the negative interaction (e.g., the timeout isreduced or executed immediately). With this approach, temporal elementsmay be preserved when contextual signals (e.g., collected/receivedsensor data) justify their preservation.

Referring to FIG. 1 , in some examples, a speech environment 100includes a user 10 speaking an utterance 20 within an audible range of avoice-enabled device 110 (also referred to as a device 110 or a userdevice 110) executing a digital assistant interface 120. Here, theutterance 20 spoken by the user 10 may be captured by the device 110 inthe streaming audio 12 and may correspond to a query 22 to perform anaction, or more specifically, a query 22 for the digital assistantinterface 120 to perform the action. The user 10 may prefix the query 22with a hotword 24 (e.g., invocation phrase) to trigger the device 110from a sleep or hibernation state when a hotword 24 is detected in thestreaming audio 12 by a hotword detector (e.g., the soft acceptor 200)running on the device 110 while in the sleep or hibernation state. Theuser 10 may also endpoint the query 22 with the hotword 24. Othertechniques may be used to trigger the device 110 from a sleep orhibernation state other than speaking a hotword 24. For instance,invocation events may trigger the device 110 from the sleep orhibernation state. Invocation events may include, without limitation,the user providing a gesture before or while speaking the query 22, theuser facing the device when speaking the query 22, the user suddenlyentering the presence of the device 110 and speaking the query 22, orcontextual cues indicating a likelihood that speech (e.g., a query 22)is expected to be directed toward the device 110. The action may also bereferred to as an operation or task. In this sense, the user 10 may haveconversational interactions with the digital assistant interface 120executing on the voice-enabled device 110 to perform computingactivities or to find answers to questions.

The device 110 may correspond any computing device associated with theuser 10 and capable of capturing audio from the environment 100. In someexamples, user devices 110 include, but are not limited to, mobiledevices (e.g., mobile phones, tablets, laptops, e-book readers, etc.),computers, wearable devices (e.g., smart watches), music players,casting devices, smart appliances (e.g., smart televisions) and internetof things (IoT) devices, remote controls, smart speakers, etc. Thedevice 110 includes data processing hardware 112 and memory hardware 114in communication with the data processing hardware 112 and storinginstructions, that when executed by the data processing hardware 112,cause the data processing hardware 112 to perform one or more operationsrelated to speech processing.

The device 110 further includes an audio subsystem 116 with an audiocapturing device 116, 116 a (e.g., an array of one or more microphones)for capturing and converting audio within the speech environment 100into electronic signals (e.g., audio data 14). While the device 110implements the audio capturing device 116 a (also referred to generallyas a microphone 116 a) in the example shown, the audio capturing device116 a may not physically reside on the device 110, but be incommunication with the audio subsystem 116 (e.g., peripherals of thedevice 110). For example, the device 110 may correspond to a vehicleinfotainment system that leverages an array of microphones positionedthroughout the vehicle. In another example, the audio capturing device116 a may reside on another device in communication with the user device110 that is to perform the action. Additionally, the audio subsystem 116may include a playback device 116, 116 b (e.g., such as one or morespeakers 116 b) for playing back audio generated/output by the userdevice 110 (e.g., synthetic audio, synthetic speech, or audio relatingto various types of media).

The device 110 may also include a display 118 to display graphical userinterface (GUI) elements (e.g., a graphical user-interface element 202)and/or graphical content. Some examples of GUI elements include windows,screens, icons, menus, etc. For example, the device 110 may load orlaunch applications (local or remote applications) that generate GUIelements (e.g., such as the GUI element 202) or other graphical contentfor the display 118. Moreover, the elements generated in the display 118may be selectable by the user 10 and also serve to provide some form ofvisual feedback to processing activities and/or operations occurring onthe device 110. For example, the element(s) represent an action that thedevice 110 is performing or suggesting to perform in response to a query22 from the user 10. Furthermore, since the device 110 is avoice-enabled device 110, the user 10 may interact with elementsgenerated on the display 118 using various voice commands as well asother types of commands (e.g., gesture commands or touch inputcommands). For instance, the display 118 may depict a menu of optionsfor a particular application and the user 10 may use the interface 120to select an option through speech or other means of feedback (e.g.,tactile input, motion/gesture input, etc.). When the user 10 speaks toselect a presented option, the device 110 may be operating in a reducedspeech recognition state or using a warm word model associated with thedevice 110 to determine if particular phrases are spoken to select thepresented option. As an example, a warm word model operates to detectbinary speech (e.g., “yes” or “no”) during the time (e.g., during atimeout window) when an option is being presented by the device 110(e.g., on the display 118 of the device 110).

The speech-enabled interface (e.g., a digital assistant interface) 120may field the query 22 or the command conveyed in the spoken utterance20 captured by the device 110. The speech-enabled interface 120 (alsoreferred to as interface 120 or an assistant interface 120) generallyfacilitates receiving audio data 14 corresponding to an utterance 20 andcoordinating speech processing on the audio data 14 or other activitiesstemming from the utterance 20. The interface 120 may execute on thedata processing hardware 112 of the device 110. The interface 120 maychannel audio data 14 that includes an utterance 20 to various systemsrelated to speech processing or query fulfillment.

Furthermore, the device 110 is configured to communicate via a network130 with a remote system 140. The remote system 140 may include scalableremote resources 142, such as remote data processing hardware 144 (e.g.,remote servers or CPUs) and/or remote memory hardware 146 (e.g., remotedatabases or other storage hardware). The device 110 may utilize theremote resources 142 to perform various functionality related to speechprocessing (e.g., by the speech processing system 150) and/or statepreservation of a GUI element (e.g., by the preserver 200). Forinstance, the device 110 is configured to perform speech recognitionusing a speech recognition system 152 and/or speech interpretation usinga speech interpreter 154. In some examples, although not shown, thedevice 110 may additionally convert text-to-speech (TTS) during speechprocessing using a TTS system.

The device 110 is also configured to communicate with the speechprocessing system 150. The speech processing system 150 is generallyable to perform various functionality related to speech processing suchas speech recognition and speech interpretation (also known as queryinterpretation). For instance, the speech processing system 150 of FIG.1 is shown to include a speech recognizer 152 that performs automatedspeech recognition (ASR), a speech interpreter 154 that determines themeaning of the recognized speech (i.e., to understand the speech), and asearch engine 156 to retrieve any search results in response to a queryidentified in the recognized speech. When a hotword detector (e.g.,associated with the assistant interface 120) detects a hotword event,the hotword detector passes the audio data 14 to the speech processingsystem 150. The hotword event indicates that the hotword detectoraccepts a portion of the audio data 14 (e.g., the first audio segment)as a hotword 24. With a portion of the audio data 14 identified as ahotword 24, the hotword detector and/or the assistant interface 120communicates the audio data 14 as a hotword event such that the speechprocessing system 150 can perform speech processing over the audio data14. By performing speech processing over the audio data 14, the speechrecognizer 152 in combination with the speech interpreter 154 is able todetermine whether a second audio segment (e.g., shown as the query 22)of the audio data 14 is indicative of a spoken query-type utterance.

A speech recognizer 152 receives audio data 14 corresponding to ahotword event as input and transcribes the audio data 41 into atranscription as an output referred to as a speech recognition result R.Generally speaking, by converting audio data 14 into a transcription,the speech recognizer 152 allows the device 110 to recognize when aspoken utterance 20 from the user 10 corresponds to a query 22 (orcommand), or some other form of audio communication. The transcriptionrefers to a sequence of text that the device 110 (e.g., the assistantinterface 120 or the speech processing system 150) may then use togenerate a response to the query or the command. The speech recognizer152 and/or the interface 120 may provide the speech recognition result Rto the speech interpreter 154 (e.g., a natural language understand (NLU)module) to perform semantic interpretation on the result R to determinewhether the audio data 14 includes a query 22 requesting a particularaction 158 to be performed. In other words, the speech interpreter 154generates an interpretation I of the result R in order to identify thequery 22 or command in the audio data 14 and to allow the speechprocessing system 150 to respond to the query 22 with a correspondingaction 158 invoked by the query 22. For instance, if the query 22 is acommand to play music, the corresponding action 158 invoked by the query22 is to play the music (e.g., by executing an application that iscapable of playing music). In some examples, the speech processingsystem 150 employs a search engine 156 to retrieve search results thatenable the speech processing system 150 to respond to the query 22(i.e., fulfill the query 22).

The user device 110 also includes or is associated with a sensor system160 configured with sensors 162 to capture sensor data 164 within theenvironment of the user device 110. The user device 110 maycontinuously, or at least during periodic intervals, receive the sensordata 164 captured by the sensor system 160 to determine a current state16 of the user 10 of the user device 110. Some examples of sensor data164 include motion data, image data, connection data, noise data, speechdata, or other data indicative of a state 16 of the user 10/user device110 or state of the environment in the vicinity of the user device 110.Motion data may include accelerometer data that characterizes movementof the user 10 via movement of the user device 110. For instance, theuser 10 is holding the device 110 and, when the user 10 moves her thumbto tap the display 118 of the device 110 (e.g., to interact with a GUIelement), the motion data indicates that the state 16 of the user 10 isengaging with the device 110. Motion data could also be received at thedevice 110 from another device associated with the user such as smartphone in the user's pocket or a smart watch worn by the user. Image datamay be used to detect an attention of the user 10, a proximity of theuser 10 with respect to the user device 110 (e.g., a distance betweenthe user 10 and the user device 110), a presence of the user 10 (e.g.,is the user 10 present or not within a field of view of one or moresensor 162), and/or features of the environment of the user 10. Forexample, the image data detects the attention of the user 10 bycapturing features of the user 10 (e.g., to characterize a gesture ofthe user 10 by body features, to characterize a gaze of the user 10 byfacial features, or to characterize a pose/orientation of the user 10).Connection/communication data may be used to determine whether the userdevice 110 is connected to or in communication with other electronics ordevices (e.g., a smart watch or a mobile phone). For instance, theconnection data may be near-field connection/communication data,Bluetooth connection/communication data, Wi-Fi connection/communicationdata, or some other radio band connection/communication data (e.g.,ultra-wideband (UWB) data). Acoustic data, such as noise data or speechdata, may be captured by the sensor system 160 (e.g., a microphone 116 aof the device 110) and used to determine the environment of the userdevice 110 (e.g., characteristics or properties of the environment thathave particular acoustic signatures) or identify whether the user 10 oranother party is speaking. In some configurations, the sensor system 160captures ultra-sonic data to detect a location of the user 10 or otherobjects within the environment of the device 110. For instance, thedevice 10 leverages the combination of its speaker(s) 116 b and itsmicrophone(s) 116 a to capture ultra-sound data for the environment ofthe device 110. The sensor(s) 162 of the sensor system 160 may beembedded or hosted on-device (e.g., a camera that captures image data ora microphone 116 a that captures acoustic data), reside off-device, butin communication with the device 110, or some combination thereof.Although FIG. 1A depicts a camera included on the device 110 as anexample sensor 162, other peripherals of the device 110 may alsofunction as sensors 162, such as the microphone 116 a and the speaker116 b of the audio subsystem 116.

Systems 150, 160, 200 may reside on the device 110 (referred to ason-device systems) or reside remotely (e.g., reside on the remote system140), but in communication with the device 110. In some examples, someof these systems 150, 160, 200 reside locally or on-device while othersreside remotely. In other words, any of these systems 150, 160, 200 maybe local or remote in any combination. For instance, when a system 150,160, 200 is rather large in size or processing requirements, the system150, 160, 200 may reside in the remote system 140. Yet when the device110 may support the size or the processing requirements of one or moresystems 150, 160, 200, the one or more systems 150, 160, 200 may resideon the device 110 using the data processing hardware 112 and/or thememory hardware 114. Optionally, the one or more of the systems 150,160, 200 may reside on both locally/on-device and remotely. Forinstance, one or more of the systems 150, 160, 200 may default toexecute on the remote system 140 when a connection to the network 130between the device 110 and remote system 140 is available, but when theconnection is lost or the network 130 is unavailable, the systems 150,160, 200 instead execute locally on the device 110.

The preserver 200 generally functions as a system to dynamically adapt astate S of a GUI element (also referred to as a user-interface (UI)element) being displayed on the user device 110 (e.g., a user interfacesuch as the display 118 of the device 110). The state S of a GUI elementbroadly refers to properties of the GUI element. That is, the state S ofthe GUI element may refer to a location of the GUI element, an amount oftime that the GUI element is presented (i.e., the timeout value), and/orcharacteristics of graphics associated with the GUI element (e.g.,color, typeface, font, style, size, etc.). In this respect, changing thestate S of the GUI element includes, for example, changing a size of theGUI element, changing the location of the GUI element within thedisplay, changing the time that the GUI element is presented, changingthe font of the GUI element, changing the content of the GUI element(e.g., changing the text or media content of the GUI element), etc.Although the preserver 200 is capable of adapting the state S of any GUIelement, the examples herein more specifically illustrate the preserver200 changing the state S of a temporal GUI element 202. A temporal GUIelement 202 is a graphical element that is temporary in nature. Forexample, the temporal GUI element 202 includes a timeout value whichdesignates a time T that the temporal GUI element will exist (e.g., bedisplayed) before being removed or automatically dismissed.

While examples herein refer to a temporal GUI element displayed on adisplay 118, implementations herein are equally applicable to presentinga temporal element through non-graphical interfaces such as flashing alight and/or audible output of a sound (e.g., beep/chime). The user mayinteract with the this type of non-graphical interface element 202 viapressing a physical button on the device, providing a voice input, orperforming a gesture.

To identify if and when the preserver 200 should change (i.e.,dynamically adapt) the state S of the temporal GUI element 202 beingdisplayed on the user device 110, the preserver 200 is configured toreceive or to monitor contextual signals 204 accessible to the device110. In some examples, the sensor data 164 captured by the sensor system160 serves as one or more contextual signals 204 characterizing aspectsof the device's environment. The preserver 200 may use the sensor data164 as contextual signals 204 without any further processing of thesensor data 164 or perform further processing on the sensor data 164 togenerate the contextual signals 204 that characterize aspects of thedevice's environment. The device 110 may leverage the preserver 200 todetermine whether a contextual signal 204 indicates that the user 10 istrying to interact with the temporal GUI element 202 prior to itsextinction. That is, a contextual signal 204 is capable of indicating astate 16 of the user 10 where the state 16 is indicative of anengagement state where the user 10 is trying to interact with thetemporal GUI element 204 or a disengagement state where the user 10 isnot interacting or purposefully disengaging with the temporal GUIelement 204. Depending on the state 16 of the user 10 characterized bythe contextual signal 204, the preserver 200 may either preserve thetemporal GUI element 202 for a longer period of time than originallydesignated, advance the extinction (i.e., removal) of the temporal GUIelement 202, or maintain the time-based properties (e.g., the timeoutvalue) of the temporal GUI element 202 (i.e., do not change any state Sof the temporal GUI element 202).

Returning to the first grocery example, when the user 10 is movingtoward the device 110, the preserver 200 may receive one or morecontextual signals 204 that characterize the state 16 of the user 10 asan engagement state 16, 16 e. For instance, the contextual signals 204indicate that the proximity of the user 10 with respect to the userdevice 110 is changing in a manner that is making the user 10 moreproximate to the user device 110 (i.e., the distance between the user 10and the user device 110 is decreasing). Because the contextual signals204 indicate that the state 16 of the user 10 is changing, this suggeststhe user 10 is trying to interact with the prompt being displayed on theuser device 110. Here, the preserver 200 modifies the state S of theprompt “do you want to play the local radio station,” which is atemporal GUI element 202. In this example, the preserver 200 wouldmodify the state S of the prompt by extending or suspending the timeoutduration for the prompt to enable to the user 10 to successfullyinteract with the prompt before it expires. Contextual signals 204 mayindicate any action performed by the user that could characterize thestate 16 of the user 10 in the engagement state 16 e. As anotherexample, the user 10 picking up a remote control could characterize thestate of the user 10 is in the engagement state 16 to interact with atemporal GUI element 202 displayed on a television.

In contrast, when the user 10 is talking on the phone and did notcommand the device 110 to play the local radio station, the preserver200 may receive one or more contextual signals 204 that characterize thestate 16 of the user 10 as a disengagement state 16 d. For example, asdescribed in this version of the example, the user 10 stares at thedevice 110 displaying the prompt “do you want to play the local radiostation” and then proceeds to turn and continue bringing in groceries.When these are the actions of the user 10, the preserver 200 may receivecontextual signals 204 that characterize an attention of the user 10 asengaged with, or staring at the device 110. These contextual signals 204are then followed by contextual signals 204 that characterize adisengagement with the device 110 (i.e., turning away from the device110) without any action by the user 10 that indicates an attempt toengage with the temporal GUI element 202. In fact, the contextualsignals 204 received by the preserver 200 would then indicate that theuser 10 is decreasing her proximity with respect to the device 110indicative of a disengagement state 16 d. With these changing contextualsignals 204 that overall indicate an engagement state 16 e followed by adisengagement state 16 d, the preserver 200 may either maintain thestate S of the temporal GUI element 202 (i.e., let the prompt expire),advance the state S of the temporal GUI element 202 (i.e., shorten thetimeout duration or remaining timeout duration), or change the S of thetemporal GUI 202 to remove the temporal GUI element 202 immediately. Inthis example, the preserver 200 may interpret the contextual signals 204as characterizing a deliberate disengagement due to a change from anengagement state 16 e (e.g., a state 16 that recognizes the temporal GUIelement 202) to a disengagement state 16 d and promptly remove thetemporal GUI element 202 once this interpretation is made.

Referring to FIGS. 2A and 2B, the preserver 200 includes a statedeterminer 210 and a modifier 220. The state determiner 210 isconfigured to receive one or more contextual signals 204 and todetermine a state 16 of a user 10 in relation to the device 110. In someexamples, the state determiner 210 receives the sensor data 164 (e.g.,raw sensor data) from the sensor system 160 and converts or processesthe sensor data 164 into contextual signals 204 that characterize thestate 16 of the user 10. For instance, the state determiner 210 receivesthe sensor data 164 and may characterize a physical state of the user 10(e.g., a position or an orientation of the user 10) relative to thedevice 110 based on the sensor data 164 in order to form the contextualsignal 204. Additionally or alternatively, the state determiner 210receives the sensor data 164 and may characterize a non-physical state16 of the user 10. For example, the state determiner 210 receivesacoustic data 164 that characterizes the state 16 of the user 10. Toillustrate, the state determiner 210 receives acoustic data 164 thatindicates that the user 10 was speaking and paused or slowed her speechin response to the device 110 displaying the temporal GUI element 202.In other words, the user 10 may been having a side speech conversationwith another user and noticed that the device 110 displayed a prompt asa temporal GUI element 202. Due to the user 10 noticing the prompt, theuser 10 may have slowed her speech in the conversation briefly (i.e.,paused momentarily to read or view the prompt) and then continued tocarry on in the conversation ignoring the prompt. In this situation, thechange in prosody of the acoustic data 164 may be a contextual signal204 that indicates the state 16 of the user 10. Here, in thisillustration, by the user 10 changing the prosody (e.g., rhythm) of herspeech, the state determiner 210 may determine that the user 10 is in anengaging state 16 e. Then, when the user 10 changes back to the originalprosody, the state determiner 210 may determine that the user 10 is in adisengaging state 16 d.

In some examples, the contextual signal 204 refers to a perceived changein state 16 of the user 10. That is, the state determiner 210 identifiessensor data 164 from a first instance in time and compares that thesensor data 164 from the first instance in time to sensor data 164 froma second instance in time to determine whether the comparison of sensordata 164 indicates a state change (e.g., a change in location ororientation) for the user 10. Due to the perceived change in state 16for the user 10, the state determiner 210 may identify whether thechange in state 16 indicates that the user 10 is trying to interact withthe device 110 in some manner (i.e., engage with the device 110 byinteracting with the temporal GUI element 202) or not interact with thedevice 110. When the change in state 16 for the user 10 indicates thatthe user 10 is trying to interact with the device 110, the user 10 isthen in an engagement state 16 a. On the other hand, when the change instate 16 for the user 10 indicates that the user 10 is trying to notinteract with the device 110, the user 10 is then in a disengagementstate 16 d. In some configurations, the state determiner 210 mayclassify the state 16 of the user 110 with greater granularity thanengaged or disengaged. For instance, the state determiner 210 isconfigured to classify the type of user engagement (e.g., approaching,gesturing in affirmation, speaking confirmation, etc.) or the type ofuser disengagement (e.g., leaving, gesturing in negation, speakingnegatively, etc.)

Once the state determiner 210 identifies the state 16 of the user 10,the state determiner 210 passes the state 16 to the modifier 220. Themodifier 220 is configured to modify the state S of the temporal GUIelement 202 being displayed on the device 110 based on the state 16 ofthe user 10. That is, the modifier 220 is able to allow the state S ofthe temporal GUI element 202 to adapt to the state 16 of the user 10while the temporal GUI element is being displayed on the device 110.Referring to FIG. 2A, the modifier 210 may change the state S of thetemporal GUI element 202 from a first state S, S₁ to a second state S,S₂ or decide that state S of the temporal GUI element 202 should notchange based on the state 16 of the user 10 (e.g., remain the firststate S₁). The modifier 220 may be configured to change the state S ofthe temporal GUI element 202 in different ways. In some examples, themodifier 220 modifies or changes the state S of the temporal GUI element202 by increasing a time (or the time remaining) that the temporal GUIelement 202 will be displayed. FIG. 2B depicts that the modifier 220 mayincrease the timeout value from a time T of seven seconds to a time T offifteen seconds when the user state 16 is an engagement stage 16 e. Inother examples, when the user state 16 is an engagement state 16 e, themodifier 210 modifies the state S of the temporal GUI element 202 bysuspending the timeout feature of the temporal GUI element 202. That is,while the preserver 200 perceives that the user 10 is engaged with thedevice 110, the modifier 220 enables the temporal GUI element 202 toexist for an indefinite duration (or until the user state 16 changes toa disengaged state 16 d).

As also shown in FIG. 2B, the modifier 220 may modify the state S of thetemporal GUI element 202 when the user state 16 is a disengaged state 16d. For instance, when the user 10 is in a disengaged state 16 d, themodifier 220 accelerates or advances the time remaining until thetemporal GUI element 202 times out. Here, FIG. 2B illustrates twosituations where this may occur. In the first scenario, when the userstate 16 is a disengaged state 16 d, the modifier 220 responds byexpiring the temporal GUI element 220 on the basis that the user 10 isnot interested in engaging with the temporal GUI element 220. This mayoccur when the user state 16 changes from an engaged state 16 e to adisengaged state 16 d while the temporal GUI element 202 is beingdisplayed much like the second grocery example where the user 10 staresat the device 110 and then walks away. FIG. 2B illustrates this firstscenario by depicting the timeout time T of seven seconds in the firststate S₁ changing to a timeout time T of zero seconds in the secondstate S₂. In the second scenario, when the user state 16 is a disengagedstate 16 d, the modifier 220 responds by advancing the time T until thetemporal GUI element 220 expires. For example, the modifier 220 changesthe timeout time T from seven seconds to three seconds.

In some configurations such as FIG. 2A, the preserver 200 additionallyincludes a state monitor 230. The state monitor 230 may be configured tomonitor and/or regulate a number of times that the state 16 of the user10 impacts the state S of the temporal GUI element 202. To illustratewhy this may be advantageous, the user 10 in the grocery example mayfinish bringing the groceries in from the car and start putting thegroceries away. During this time, the user 10 may be moving back andforth in the kitchen such that the state determiner 210 perceives thatthe user 10 is engaging and disengaging with the device 110 while thedevice 110 displays the temporal GUI element 202. Due to this engagementand disengagement, the modifier 220 may be changing the state S of thetemporal GUI element 202. For instance, the modifier 220 may beincreasing the timeout duration, then decreasing the timeout duration,in some repetitive fashion that mirrors the user's actions.Unfortunately, when this occurs some number of times, it more likelyindicates that the temporal GUI element 202 should not be dynamicallychanged, but rather expired or left to expire because the user 10 wouldhave selected or interacted with the temporal GUI element 202 already.To prevent oscillating state changes or too many state changes fromoccurring, the monitor 230 may be configured with a state changethreshold 232. In some examples, when the number of state changes forthe temporal GUI element 202 satisfies the state change threshold 232,the monitor 230 deactivates the modifier 220 with respect to thetemporal GUI element 202 or otherwise allow/force the temporal GUIelement 202 to expire (e.g., timeout). As the monitor 230 is operating,if the number of state changes for the temporal GUI element 202 does notsatisfy the state change threshold 232, the monitor 230 does not preventthe modifier 220 from modifying the temporal GUI element 202 (e.g., themodifier 220 continues to operate).

In some configurations, the monitor 230 is able to adjust one or morethresholds associated with the state determiner 210. For instance, thestate determiner 210 may identify a state change for the user 10 basedon the user 10 moving from a far field location to a near field locationwith respect to the device 110. Here, the far field and near field maybe delineated by a threshold (e.g., shown in FIG. 3 as threshold 212)that establishes a proximity boundary between the near field and farfield. In other words, if the user 10 crosses the threshold movingtowards the device 110, the user has entered the near field (e.g., leftthe far field) while if the user 10 crosses the threshold moving awayfrom the device 110, the user has entered the far field (e.g., left thenear field). Yet there are circumstances where the user 10 may be movingback and forth at this boundary. To illustrate, the user 10 may beputting away the groceries from a kitchen island into a fridge and theboundary may be located in between the kitchen island and the fridge.Since the user 10 is likely to be constantly traveling back and forthover this boundary, the monitor 230 may recognize this activity at thethreshold and function as a type of debouncer. That is, the user 10 mayonly be moving a few feet and instead of allowing the state determiner210 to identify this movement as changing between the engagement state16 e and the disengagement state 16 d, the monitor 230 adjusts thethreshold to stabilize the state change sensitivity of the statedeterminer 210. In other words, the user 10 would have to travel furthertoward the device 110 or away from the device than simply the island totrigger a change of state 16.

FIG. 3 is a scenario that illustrates some types of contextual signals204 that may characterize the state 16 of the user 10 (e.g., thephysical state 16 of the user) at a first time instance T₀ and a secondtime instance T₁ that occurs after the first time instance T₀. In thisscenario, the user 10 walks into the room with the device 110 talking onher mobile phone saying, “I just heard this great song playing on thelocal radio station, 94.5.” In response to this utterance 20 by the user10, the device 110 generates a prompt as a temporal GUI element 202 thatinquires whether the user 10 wants to play the local radio station 94.5.Here, the preserver 200 receives sensor data 164 that corresponds toimage data at the first time instance T₀ depicting a location of theuser 10 relative to the user device 110. With the image data, the statedeterminer 210 estimates a distance D, D₁ between the user 10 and thedevice 110 at the first time instance T₀ based on the received imagedata to form a contextual signal 204 that characterizes the state 16 ofthe user 10 in relation to the device 110. Here, a contextual signal 204that indicates the user's proximity to the device 110 is considered auser-proximity signal. At the first time instance T₀, the statedeterminer 210 identifies that the estimated distance D, D₁ means thatthe user 10 is in the near field at a proximity to the device 110 withinthe near-far boundary 212. For instance, the state determiner 210determines that the estimated distance D₁ has a distance D to the device110 that is less than the distance from the device 110 to the boundary212. Based on the user's proximity to the device 110, the statedeterminer 210 determines that the user 10 is in an engagement state 16e at the first time instance T₀. Since the user-proximity signalindicates that the proximity of the user 10 relative to the device 10has changed to be closer to the device 10 (e.g., the user 10 walkstoward the device 10 in FIG. 3 to be at a first estimated distance D₁),the modifier 220 may then modify the state S of the temporal GUI element202 by, for example, increasing a timeout duration or suspending thetimeout duration of the temporal GUI element 202.

Notably, while the temporal GUI element 202 prompt is displayed, thedevice 110 may initiate fulfillment of the perceived command to play thelocal radio station 94.5 by initiating a connection with station 94.5(or a music streaming service capable of streaming station 94.5) andstreaming audio therefrom. However, the device 110 may not begin toaudibly output the streaming audio until the user 10 affirmativelyprovides the user input indication indicating selection of the temporalGUI element 202 to stream the audio. Accordingly, in this example, sincethe user 10 did not direct the speech to the device 110 to stream audiofrom station 94.5, the device 110 would terminate the streamingconnection in response to the temporal GUI element 202 timing out (orthe user affirmatively providing an input indication selecting “No”)without ever audibly outputting the streaming audio.

Additionally, while examples herein are directed toward the user device110 generating prompts as temporal GUI elements 202, the user device 110may similarly output a visual (e.g., light flash) and/or audible (e.g.,beep) prompt in a non-obtrusive manner that allows the user to affirm ordeny performance of the recognized voice command within the temporaltimeout period. For instance, the user 10 could simply speak “yes” or“no”. In this scenario, the user device 110 could activate a warm wordmodel that listens for binary terms (e.g., “yes” and “no”) in audio.

In addition to allowing the state determiner 210 to generate auser-proximity signal for the contextual signal 202, the sensor data 164in this scenario also allows the state determiner 210 to generate acontextual signal 204 referred to as an attention detection signal. Anattention detection signal refers to a signal that characterizes whetherthe user 10 is attending to the device 110. An attention detectionsignal may indicate whether the user 10 is in an engagement state 16 eor a disengagement state 16 d. In other words, an attention detectionsignal (e.g., based on sensor data 164) may indicate that the user 10has changed his or her focus either toward the device 110 or away fromthe device 110. When the focus of the user 10 changes toward the device110 while the device 110 displays the temporal GUI element 202, thestate determiner 210 may determine that the user 10 is in an engagementstate 16 e. Some examples that the user 10 is attending to the device110 include a gaze of the user 10 directed to the device 110, a gestureof the user 10 directed toward the device 110, or a pose/orientation ofthe user 10 directed toward the device 110 (i.e., faces the device 110).Referring to FIG. 3 , the image data captures a gaze of the user 10 thatis directed to the device 110 (e.g., shown as dotted vision lines from aface of the user 10). With this attention detection signal 204indicating that the user 10 is focusing her attention on the device 110,the state determiner 210 determines that the attention detection signal204 characterizes that the user 10 is in an engagement state 16 e at thefirst time instance T₀. Since the attention detection signal aloneindicates that the attention of the user 10 is focused on the device 10,the modifier 220 may then modify the state S of the temporal GUI element202 by, for example, increasing a timeout duration or suspending thetimeout duration of the temporal GUI element 202. In some examples, suchas this scenario, when multiple contextual signals 204 are available,the preserver 200 may leverage any or all of the contextual signals 204to determine if the state 16 of the user 10 should impact (e.g., modify)the state S of the temporal GUI element 202.

FIG. 3 also illustrate that the user 10 moves from the first location ata first distance D₁ from the device 110 to a second location at a seconddistance D₂ from the device 110 at a second time instance T₁. At thesecond time instance T₁, the contextual signals 204 indicate that theuser 10 is in a disengaging state 16 d where the user 10 does not intendto interact with the temporal GUI element 202. For instance, at thesecond time instance T₁, a user-proximity signal indicates that the user10 relative to the device 110 has changed to be further from the device110 and cross the boundary 212 to be in the far field. Furthermore, atthe second time instance T₁, an attention detection signal indicatesthat the user 10 has diverted her attention from the device 110. Sinceboth of the contextual signals 204 indicate that the user 10 is in adisengaging state 16 d, the modifier 220 may modify the state S of thetemporal GUI element 202 (e.g., by advancing or expiring the temporalGUI element 202) or maintain the state S of the temporal GUI element 202to let it expire accordingly.

Although not shown, instead of the user 10 being located in the farfield at the second time instance T₁, the user 10 may have left the roomcompletely. In this situation, the state determiner 210 may generate acontextual signal 204 referred to as a presence detection signal. Apresence detection signal is a signal that characterizes whether theuser 10 is present or not in a particular field of view for one or moresensors 162. In other words, the presence detection signal may functionas a binary determination of whether image data or some other form ofsensor data 164 indicates the user 10 is present within a field of viewof a sensor 162 of the device 110. A presence detection signal may beindicative of a user's state because a change in the presence detectionsignal may indicate whether the user 10 is performing an action thatcharacterizes the user 10 to be in an engagement state 16 e or adisengagement state 16 d. For instance, if the user 10 was not presentin a field of view of the device 110 (e.g., of a sensor 162 associatedwith the device 110) and then becomes present in the field of view ofthe device 110 and this change happens while the temporal GUI element202 is being displayed on the display 118 of the user device 110, thestate determiner 210 may interpret that the change in the user'spresence is indicative of the user trying to interact with the temporalGUI element 202. That is, the user 10 has become present to engage withthe temporal GUI element 202 such that the state 16 of the user 10 is anengagement state. In the reverse situation, if the user 10 was initiallypresent in a field of view of the device 110 (e.g., of a sensor 162associated with the device 110) and then is no longer present in thefield of view of the device 110 and this change happens while thetemporal GUI element 202 is being displayed on the display 118 of theuser device 110, the state determiner 210 may interpret that the changein the user's presence is indicative of the user uninterested ininteracting with the temporal GUI element 202 (e.g., activelydisengaging) such that the state 16 of the user 10 is a disengagementstate 16 d. When the presence detection signal indicates that thepresence of the user 10 within the field of view of the sensor 162 haschanged from not present to present, the modifier 220 may then modifythe state S of the temporal GUI element 202 by, for example, increasinga timeout duration or suspending the timeout duration of the temporalGUI element 202. In contrast, when the presence detection signalindicates that the presence of the user 10 within the field of view ofthe sensor 162 has changed from present to not present, the modifier 220may then modify the state S of the temporal GUI element 202 by, forexample, decreasing a timeout duration, immediately expiring thetemporal GUI element 202, or maintaining the timeout duration of thetemporal GUI element 202 to allow it to expire accordingly.

FIG. 4 is a flowchart of an example arrangement of operations for amethod 400 of changing a state of a graphical user-interface element202. The method 400 performs operations 402-406 in response to detectingthat a temporal GUI element 202 displayed on a user interface of a userdevice 110. At operation 402, the method 400 receives, at the userdevice 110, a contextual signal 204 characterizing a state 16 of theuser 10. At operation 404, the method 400 determines, by the user device110, that the contextual signal 204 characterizing the state 16 of theuser 10 is indicative of the user intending to interact with thetemporal GUI element 202. In response to determining that the contextualsignal 204 characterizing the state 16 of the user 10 is indicative ofthe user intending to interact with the temporal GUI element 202, atoperations 406, the method 400 modifies a respective state S of thetemporal GUI element 202 displayed on the user interface of the userdevice 110.

FIG. 5 is a schematic view of an example computing device 500 that maybe used to implement the systems (e.g., the systems 150, 160, 200) andmethods (e.g., method 400) described in this document. The computingdevice 500 is intended to represent various forms of digital computers,such as laptops, desktops, workstations, personal digital assistants,servers, blade servers, mainframes, and other appropriate computers. Thecomponents shown here, their connections and relationships, and theirfunctions, are meant to be exemplary only, and are not meant to limitimplementations of the inventions described and/or claimed in thisdocument.

The computing device 500 includes a processor 510 (e.g., data processinghardware 112, 144), memory 520 (e.g., memory hardware 114, 146), astorage device 530, a high-speed interface/controller 540 connecting tothe memory 520 and high-speed expansion ports 550, and a low speedinterface/controller 560 connecting to a low speed bus 570 and a storagedevice 530. Each of the components 510, 520, 530, 540, 550, and 560, areinterconnected using various busses, and may be mounted on a commonmotherboard or in other manners as appropriate. The processor 510 canprocess instructions for execution within the computing device 500,including instructions stored in the memory 520 or on the storage device530 to display graphical information for a graphical user interface(GUI) on an external input/output device, such as display 580 coupled tohigh speed interface 540. In other implementations, multiple processorsand/or multiple buses may be used, as appropriate, along with multiplememories and types of memory. Also, multiple computing devices 500 maybe connected, with each device providing portions of the necessaryoperations (e.g., as a server bank, a group of blade servers, or amulti-processor system).

The memory 520 stores information non-transitorily within the computingdevice 500. The memory 520 may be a computer-readable medium, a volatilememory unit(s), or non-volatile memory unit(s). The non-transitorymemory 520 may be physical devices used to store programs (e.g.,sequences of instructions) or data (e.g., program state information) ona temporary or permanent basis for use by the computing device 500.Examples of non-volatile memory include, but are not limited to, flashmemory and read-only memory (ROM)/programmable read-only memory(PROM)/erasable programmable read-only memory (EPROM)/electronicallyerasable programmable read-only memory (EEPROM) (e.g., typically usedfor firmware, such as boot programs). Examples of volatile memoryinclude, but are not limited to, random access memory (RAM), dynamicrandom access memory (DRAM), static random access memory (SRAM), phasechange memory (PCM) as well as disks or tapes.

The storage device 530 is capable of providing mass storage for thecomputing device 500. In some implementations, the storage device 530 isa computer-readable medium. In various different implementations, thestorage device 530 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device, a flash memory or other similarsolid state memory device, or an array of devices, including devices ina storage area network or other configurations. In additionalimplementations, a computer program product is tangibly embodied in aninformation carrier. The computer program product contains instructionsthat, when executed, perform one or more methods, such as thosedescribed above. The information carrier is a computer- ormachine-readable medium, such as the memory 520, the storage device 530,or memory on processor 510.

The high speed controller 540 manages bandwidth-intensive operations forthe computing device 500, while the low speed controller 560 manageslower bandwidth-intensive operations. Such allocation of duties isexemplary only. In some implementations, the high-speed controller 540is coupled to the memory 520, the display 580 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 550,which may accept various expansion cards (not shown). In someimplementations, the low-speed controller 560 is coupled to the storagedevice 530 and a low-speed expansion port 590. The low-speed expansionport 590, which may include various communication ports (e.g., USB,Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or moreinput/output devices, such as a keyboard, a pointing device, a scanner,or a networking device such as a switch or router, e.g., through anetwork adapter.

The computing device 500 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 500 a or multiple times in a group of such servers 500a, as a laptop computer 500 b, or as part of a rack server system 500 c.

Various implementations of the systems and techniques described hereincan be realized in digital electronic and/or optical circuitry,integrated circuitry, specially designed ASICs (application specificintegrated circuits), computer hardware, firmware, software, and/orcombinations thereof. These various implementations can includeimplementation in one or more computer programs that are executableand/or interpretable on a programmable system including at least oneprogrammable processor, which may be special or general purpose, coupledto receive data and instructions from, and to transmit data andinstructions to, a storage system, at least one input device, and atleast one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium” and“computer-readable medium” refer to any computer program product,non-transitory computer readable medium, apparatus and/or device (e.g.,magnetic discs, optical disks, memory, Programmable Logic Devices(PLDs)) used to provide machine instructions and/or data to aprogrammable processor, including a machine-readable medium thatreceives machine instructions as a machine-readable signal. The term“machine-readable signal” refers to any signal used to provide machineinstructions and/or data to a programmable processor.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby special purpose logic circuitry, e.g., an FPGA (field programmablegate array) or an ASIC (application specific integrated circuit).Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both. The essential elements of a computer area processor for performing instructions and one or more memory devicesfor storing instructions and data. Generally, a computer will alsoinclude, or be operatively coupled to receive data from or transfer datato, or both, one or more mass storage devices for storing data, e.g.,magnetic, magneto optical disks, or optical disks. However, a computerneed not have such devices. Computer readable media suitable for storingcomputer program instructions and data include all forms of non-volatilememory, media and memory devices, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto optical disks; and CD ROM and DVD-ROM disks. The processor andthe memory can be supplemented by, or incorporated in, special purposelogic circuitry.

To provide for interaction with a user, one or more aspects of thedisclosure can be implemented on a computer having a display device,e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, ortouch screen for displaying information to the user and optionally akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. Accordingly, otherimplementations are within the scope of the following claims.

What is claimed is:
 1. A computer-implemented method when executed bydata processing hardware causes the data processing hardware to performoperations comprising: receiving audio data characterizing an utterancecaptured in streaming audio by a user device executing a digitalassistant interface; processing the audio data to: detect a query in theaudio data, the query specifying an action for the digital assistantinterface to perform; and determine that a confidence of the querydetected in the audio data does not satisfy automatic performance of theaction by the digital assistant interface; in response to determiningthat the confidence of the query detected in the audio data does notsatisfy automatic performance of the action: outputting, from the userdevice, a prompt that allows a user of the user device to affirmativelyprovide, within a timeout duration, a user input indication indicatingconfirmation that the user intended the query specifying the action; andduring the timeout duration before the user affirmatively provides theuser input indication, initiating fulfillment of the query specifyingthe action; and based on determining that the user of the user devicedid not affirmatively provide the user input indication within thetimeout duration, terminating performance of the action specified by thequery.
 2. The computer-implemented method of claim 1, wherein the audiodata characterizing the utterance captured in the streaming audiocomprises a hotword that triggers the user device from a sleep orhibernation state when a hotword detector running on the user devicedetects the in the streaming audio.
 3. The computer-implemented methodof claim 2, wherein the hotword prefixes the query in the utterance. 4.The computer-implemented method of claim 2, wherein the utteranceendpoints the query with the hotword.
 5. The computer-implemented methodof claim 1, wherein outputting the prompt from the user device comprisesdisplaying a temporal user-interface element on a user interface of theuser device for the timeout duration.
 6. The computer-implemented methodof claim 5, wherein the temporal user-interface element, while displayedon the user interface of the user device for the timeout duration,allows the user to affirmatively provide the user input indication byselecting the temporal user-interface element.
 7. Thecomputer-implemented method of claim 5, wherein the operations furthercomprise, during the timeout duration before the user affirmativelyprovides the user input indication: receiving a contextual signalcharacterizing a state of the user; determining that the contextualsignal characterizing the state of the user is indicative of the userintending to interact with the temporal user-interface element; and inresponse to determining that the contextual signal characterizing thestate of the user is indicative of the user intending to interact withthe temporal user-interface element, modifying the timeout duration ofthe temporal user-interface element displayed on the user interface ofthe user device.
 8. The computer-implemented method of claim 1, whereinoutputting the prompt from the user device comprises outputting a visualprompt from the user device.
 9. The computer-implemented method of claim1, wherein outputting the prompt from the user device comprisesoutputting an audible prompt from the user device.
 10. Thecomputer-implemented method of claim 1, wherein the operations furthercomprise, in response to determining that the confidence of the querydetected in the audio data does not satisfy automatic performance of theaction, activating a warm word model during the timeout duration, theactivated warm word model configured to detect a presence of aparticular term or phrase spoken by the user to affirmatively providethe user input indication within the timeout duration.
 11. A systemcomprising: data processing hardware; and memory hardware incommunication with the data processing hardware, the memory hardwarestoring instructions that when executed on the data processing hardwarecause the data processing hardware to perform operations comprising:receiving audio data characterizing an utterance captured in streamingaudio by a user device executing a digital assistant interface;processing the audio data to: detect a query in the audio data, thequery specifying an action for the digital assistant interface toperform; and determine that a confidence of the query detected in theaudio data does not satisfy automatic performance of the action by thedigital assistant interface; in response to determining that theconfidence of the query detected in the audio data does not satisfyautomatic performance of the action: outputting, from the user device, aprompt that allows a user of the user device to affirmatively provide,within a timeout duration, a user input indication indicatingconfirmation that the user intended the query specifying the action; andduring the timeout duration before the user affirmatively provides theuser input indication, initiating fulfillment of the query specifyingthe action; and based on determining that the user of the user devicedid not affirmatively provide the user input indication within thetimeout duration, terminating performance of the action specified by thequery.
 12. The system of claim 11, wherein the audio data characterizingthe utterance captured in the streaming audio comprises a hotword thattriggers the user device from a sleep or hibernation state when ahotword detector running on the user device detects the in the streamingaudio.
 13. The system of claim 12, wherein the hotword prefixes thequery in the utterance.
 14. The system of claim 12, wherein theutterance endpoints the query with the hotword.
 15. The system of claim11, wherein outputting the prompt from the user device comprisesdisplaying a temporal user-interface element on a user interface of theuser device for the timeout duration.
 16. The system of claim 15,wherein the temporal user-interface element, while displayed on the userinterface of the user device for the timeout duration, allows the userto affirmatively provide the user input indication by selecting thetemporal user-interface element.
 17. The system of claim 15, wherein theoperations further comprise, during the timeout duration before the useraffirmatively provides the user input indication: receiving a contextualsignal characterizing a state of the user; determining that thecontextual signal characterizing the state of the user is indicative ofthe user intending to interact with the temporal user-interface element;and in response to determining that the contextual signal characterizingthe state of the user is indicative of the user intending to interactwith the temporal user-interface element, modifying the timeout durationof the temporal user-interface element displayed on the user interfaceof the user device.
 18. The system of claim 11, wherein outputting theprompt from the user device comprises outputting a visual prompt fromthe user device.
 19. The system of claim 11, wherein outputting theprompt from the user device comprises outputting an audible prompt fromthe user device.
 20. The system of claim 11, wherein the operationsfurther comprise, in response to determining that the confidence of thequery detected in the audio data does not satisfy automatic performanceof the action, activating a warm word model during the timeout duration,the activated warm word model configured to detect a presence of aparticular term or phrase spoken by the user to affirmatively providethe user input indication within the timeout duration.